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A  growing  body  of  research  attests  to  the  importance  of  complex  cognitive/concq)tual 
skills  for  successful  strategic  leadership  performance.  However,  technologies  for  assessing 
and  enhancing  these  skills  are  still  in  their  infancy,  as  is  a  comprehensive  theory  of  adult 
human  development  that  would  relate  these  skills  to  other  essential  leader  attributes. 

This  report  documents  an  investigation  of  one  approach,  an  extended  structured 
interview,  for  assessing  these  skills.  Most  of  the  academic  year  1991-92  U.S.  Army  War 
College  class  participated  in  the  effort.  The  research  provides  good  evidence  for  the  reliability 
of  the  interview,  and  suggests  that  trained  interviewers  make  consistent  judgments  concerning 
the  conceptual  skills  that  are  being  assessed.  These  findings  make  reasonable  further  efforts 
that  are  now  under  way  to  establish  the  construct  validity  of  the  interview  and  to  develop  less 
costly  approaches  to  measuring  the  same  constructs. 

This  work  was  accomplished  as  a  part  of  the  program  of  the  Strategic  Leadership 
Technical  Area  (SLTA)  or  the  Manpower  and  Personnel  Research  Division  of  the  U.S.  Army 
Research  Institute  for  the  Behavioral  and  Social  Sciences  (ARI). 


The  research  reported  herein  was  supported  by  the  U.S.  Army  Research  Institute  for 
the  Behavioral  and  Social  Sciences  (ARI)  under  the  auspices  of  the  U.S.  Army  Research 
Office  Scientific  Services  Program  administered  by  Battelle. 

Data  collection  at  the  U.S.  Army  War  College  would  not  have  been  possible  without 
the  cooperation  and  active  assistance  of  the  faculty  and  staff  of  the  Department  of  Command, 
Leadership,  and  Management  (DCLM)  and  its  Head,  Colonel  Rich  Goldsmith.  Herb  Barber 
of  DCLM  provided  particularly  invaluable  logistical  and  conceptual  support  during  the  data 
collection  phase  at  the  War  College. 

Throughout  the  project,  the  assistance,  suj^rt,  and  cooperation  of  T.  O.  Jacobs  and 
Kathie  Evans  of  ARI’s  Strategic  Leadership  Tecl^cal  Area  were  both  essratial  and 
invaluable. 
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CAREER  PATH  APPRECIATION  (CPA)  DATA  REDUCTION  AND  ANALYSIS 


EXECUTIVE  SUMM 


MX. 


Requirement: 

This  research  was  performed  to  determine  the  interrater  reliability  of  the  Career  Path 
Appreciation  (CPA)  and  to  conduct  a  psychometric  examination  of  its  component  parts.  In 
particular,  the  psychometric  examination  was  intended  to  focus  on  the  contribution  of  the 
various  parts  to  die  total  assessment,  and  on  the  internal  consistency  of  the  stimulus  items  in 
one  specific  part  of  the  total  assessment  procedure. 


Procedure: 

CPA  interviews  were  conducted  with  148  active  duty  Army  officers  in  residence  at  the 
U.S.  Army  War  College,  Carlisle  Barracks,  Pennsylvania.  Each  participant  volunteered  to  be 
interview^.  The  interviews  were  conducted  by  individuals  familiar  with  the  CPA  assessment 
procedure.  They  were  tape  recorded  and  the  tapes  were  subsequently  transcribed.  One 
hundred  and  thirty-seven  (137)  of  these  interviews  were  scored  by  the  present  investigator,  of 
which  52  were  also  scored  by  an  expert  in  Stratified  Systems  Theory.  These  rqilicated 
scorings  provided  the  basis  for  estimation  of  interrater  reliability.  In  addition,  the  stimulus 
items  in  the  PHRASES  part  of  the  CPA  were  examined  for  inteamal  consistency;  several  were 
replaced  in  response  to  this  analysis. 


Findings: 

The  interrater  reliability  of  the  CPA,  based  on  the  57  cases  scored  by  two  raters,  was 
.81.  Considering  the  quasi-clinical  nature  of  the  assessment  interview,  this  is  highly 
acceptable.  In  addition,  modest  construct  validity  was  demonstrated  for  the  CPA  Current 
Conceptual  Capacity  scores.  War  College  instructors,  who  rated  a  subsample  on  strategic 
thinking  skill  as  demonstrated  in  War  College  seminar  groups,  tended  to  rate  those  who  scored 
higher  on  the  CPA  as  better  thinkers  than  those  who  scored  lower  (R  =  .57  and  .51  on  two 
different  measures  of  thinking  skill). 

The  search  for  possible  objective  approaches  to  scoring  the  CPA  also  yielded 
promising  results.  An^ysis  of  the  PHRASES  portion  of  the  CPA  yielded  respectable 
Cronbach  Coefficient  Alphas  for  the  PHRASES  "Most"  scores  and  these  scores  were 
reasonably  good  predictors  of  rated  Current  Conceptual  Capacity  scores.  Further,  item 
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analyses  of  the  PHRASES  item  sets  suggested  that  all  nine  item  sets  should  be  retained. 
Similar  analyses  of  the  objective  indexes  generated  by  the  computer-controlled  SYMBOLS 
task  of  the  CPA  showed  that  the  index  of  the  total  number  of  cards  sorted  is  also  a  reasonably 
good  predictor  of  rated  Current  Conceptual  Capacity  for  some  raters. 

A  combination  of  the  principal  objective  scores  from  the  PHRASES  and  SYMBOLS 
tasks  (PHRASES  "Most*  average  and  SYMBOLS  total  cards  sorted),  produces  a  level  of 
prediction  of  rated  CPA  scores  that  is  very  close  to  the  reliability  estimates  of  the  measures. 
Finally,  examination  of  individual  responses  to  the  PHRASES  t^  by  subjects  classified  into 
four  conceptual  capacity  levels  on  the  basis  of  their  investigator-rated  CPA  performances 
suggested  ways  to  improve  some  of  the  PHRASES  item  sets. 


Utilization  of  Findings: 

These  findings  have  been  used  to  revise  portions  of  the  CPA  PHRASES  task  and  will 
contribute  to  further  efforts  to  develop  cost-effective  tools  for  assessing  and  facilitating  the 
development  of  the  complex  conceptual  skills  required  by  strategic  leaders. 
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CAREER  PATH  APPRECIATION  DATA  REDUCTION  AND  ANALYSIS 

INTRODUCTION 

Career  Path  Appreciation  (CPA)  is  a  complex  assessment  interview  designed  to 
measure  an  individual’s  current  capacity  for  meeting  the  conceptual  demands  of  various 
work  roles.  Developed  by  Gillian  Stamp  (1984),  the  CPA  is  based  on  two  key  concepts 
contained  in  the  Stratified  Systems  Theory  of  Elliott  Jaques  (1976,  1989).  The  first  is  the 
well-accepted  notion  that  work  is  hierarchically  stratified  (Katz  &  Kahn,  1966,  Simon, 
1977)  such  that  higher  level  managerial  positions  entail  dealing  with  greater  complexity 
than  do  lower  level  positions.  Jaques  (1976)  extended  this  general  idea  of  hierarchical 
role  complexity  by  asserting  that  a  finite  and  relatively  small  number  of  qualitative 
increases  in  work  complexity  define  the  identifiable  levels  of  vertical  work  stratification. 

The  second  key  concept  in  Jaques’  Stratified  Systems  Theory  that  informed  the 
development  of  the  CPA  was  the  idea  that  the  individual  conceptual  capacity  necessary 
to  master  work  complexity  at  the  various  organizational  levels  is  also  stratified.  That  is, 
Jaques  asserted  that  there  are  qualitatively  distinct  levels  of  individual  conceptual 
capability  that  correspond  to  levels  of  work  complexity  (Jaques,  1976,  1989;  Jaques  & 
Clement,  1991).  It  was  these  levels  of  individual  capability  that  the  CPA  was  designed  to 
assess. 


The  potential  value  to  the  military  of  a  valid  measure  of  level  of  conceptual  work 
capacity  is  considerable.  Given  the  importance  of  maintaining  highly  competent  armed 
forces,  it  is  essential  that  the  military  have  leaders  at  all  organizational  levels  who  have 
the  capability  to  understand  and  master  the  level  of  complexity  inherent  in  their  work 
roles.  For  example,  requisite  conceptual  capacity  at  upper  organizational  levels  may  be 
the  sine  qua  non  of  what  is  commonly  known  as  "strategic  vision"  (Jaques  &  Clement, 
1991;  Lewis  &  Jacobs,  1992).  In  addition,  at  all  organizational  levels  within  the  military, 
having  a  level  of  conceptual  work  capability  at  least  one  level  above  one’s  own  work  role 
may  facilitate  a  full  understanding  of  commander’s  intent. 

Because  of  its  close  conceptual  tie  to  Stratified  Systems  Theory,  the  CPA  is  a 
promising  candidate  as  a  measure  of  individual  work  capability.  Use  of  the  CPA 
assessment  methodology  requires,  however,  the  establishment  of  its  interrater  reliability 
and  construct  validity.  As  appealing  as  the  CPA  may  be  on  conceptual  grounds,  it  is  of 
little  value  unless  it  can  be  shown  to  reliably  assess  individuals’  current  levels  of 
conceptual  capability.  To  that  end,  the  present  report  provides  the  first  published 
interrater  reliability  data  for  the  CPA,  extends  its  construct  validity,  and  provides 
preliminary  suggestions  for  its  improvement. 
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TECHNICAL  DISCUSSION 


The  Career  Path  Appreciation  (CPA)  Methodology 

An  individually  administered,  multifaceted  assessment  device,  the  CPA  yields  an 
overall  score  of  current  work  capacity  and  a  related  score  of  highest  predicted  future 
work  capacity.  The  latter,  termed  "Mode"  by  Jaques  (1989),  i"  derived  from  the  former 
using  current  age  and  empirically  derived  "progression  curves."  The  CPA  consists  of 
three  sections:  Phrases,  Symbols,  and  Work  History.  In  the  Phrases  portion  of  the 
assessment,  the  interviewer  gives  the  interviewee  nine  sets  of  six  cards  where  each  card 
contains  a  phrase  (e.g.,  "Work  within  a  given  framework").  For  each  set  of  cards  the 
interviewee  is  instructed  to  read  the  six  phrases  and  then  indicate  which  phrase  is  "most" 
like  the  way  he  or  she  prefers  to  approach  a  piece  of  work,  and  which  phrase  is  "least" 
like  the  way  he/she  prefers  to  operate  at  work.  The  interviewee  is  then  asked  to 
elaborate  upon  his  or  her  choices.  And  while  the  CPA  includes  a  current  level  score  for 
each  of  the  six  phrases  in  each  set,  the  interviewee’s  elaboration  of  his  or  her  choices  is 
given  considerable  weight  in  estimating  current  conceptual  level  (CL). 

In  the  Symbols  portion  of  the  asse55ment,  the  interviewee  is  asked  to  attempt  to 
solve  a  symbol  sorting  task.  Computerized  by  the  US  Army  Research  Institute  in  1990, 
the  Symbols  task  uses  geometric  symbols  derived  from  the  Bruner  concept  formation 
task.  The  computer  program  presents  four  "target  cards,"  one  of  which  is  blank.  The 
interviewee  is  then  told  that  his  or  her  task  is  to  discover  a  sorting  rule  by  which  a  set  of 
"cards"  (computer  images)  can  be  correctly  sorted  into  piles  below  each  of  the  four  target 
cards.  Solution  of  the  sorting  task  is  considered  to  require  the  capacity  to  induce  the 
nature  of  the  required  task,  to  abstract  the  essential  attributes  of  the  symbols,  and  to  test 
complex  hypotheses  systematically. 

In  the  Work  History  portion  of  the  interview,  the  interviewee  is  asked  to  provide 
a  chronological  description  of  major  work  assignments  and  to  describe  the  level  of 
challenge  (overstretched,  comfortable,  underutilized)  and  time  span  of  discretion  in  each 
of  those  assignments.  Finally,  in  an  attempt  to  elicit  an  indicator  of  the  interviewee’s 
time  horizon,  the  interviewee  is  asked  to  describe  what  he  or  she  envisions  as  lying  in 
their  future  in  the  "near  term,"  "mid  term’"  and  "long  term." 

Scoring  the  CPA.  The  CPA,  as  used  in  the  present  investigation,  yields  a  single 
score  estimating  each  individual’s  current  conceptual  work  capacity.  These  scores  can, 
potentially,  range  from  current  level  I  to  current  level  VII.  Within  each  level,  there  are 
three  distinctions  made:  high,  medium,  or  low.  Converted  to  Arabic  numbers,  scores  can 
thus  range  from  1  (low  current  level  I)  to  21  (high  current  level  VII). 

The  present  investigator  scored  CPAs  from  three  sources  of  information:  1)  a 
paper  record  which  included  the  interviewer’s  notes  of  the  three-part  interview  and  a 
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notation  of  the  interviewee’s  age,  2)  a  typescript  of  the  Phrases  narrative,  and  3)  a 
computer  generated  record  of  the  Symbols  performance  showing  each  sorting  response 
made.  In  addition  to  assigning  an  overall  score  of  current  conceptual  work  capacity,  use 
of  the  current  level  score,  current  age,  and  Jaques’  1990  progression  curves  permitted  the 
present  investigator  to  assign  a  score  of  highest  future  potential  ("Mode").  In  accordance 
with  CPA  scoring  guidelines  (Stamp,  1984),  separate  scores  were  not  assigned  for  e«ch  of 
the  three  portions  of  the  interview.  Rather,  a  single  overall  score  was  assigned.  In 
assigning  an  overall  score  of  current  conceptual  work  capacity,  the  present  investigator 
first  examined  the  Symbols  computer  record  and  the  interviewer’s  notes  about  the 
symbols  performance  (if  there  were  any)  and  developed  a  working  hypothesis  of  current 
level.  In  general,  a  successful  Symbols  performance  was  considered  strong  evidence  that 
current  capacity  was  mid  level  IV  or  hi^er.  Weaker  successful  performances  were 
presumed  to  reflect  a  current  capacity  of  high  level  III  or,  possibly,  low  level  IV. 
Unsuccessful  performances  were  interpreted  more  equivocally,  as  there  are  undoubtedly 
multiple  reasons  for  failure  to  solve  the  Symbols  tasl^  only  some  of  which  may  reflect 
current  conceptual  capacity.  Nonetheless,  particularly  poor  performances,  in  terms  of 
the  types  of  strategies  employed,  were  usually  interpreted  as  reflecting  a  current 
capability  at  mid  III  or  below. 

After  considering  the  Symbols  performance,  the  present  investigator  evaluated  the 
Phrases  performance  as  captured  in  the  typescript  and  the  interviewer’s  notes.  Because 
of  the  nature  of  the  subjects  interviewed  for  the  present  investigation.  War  College 
students  who  had  been  successful  in  leadership  roles  at  an  organizational  level 
considered  to  be  high  level  III  (Jacobs  &  Jaques,  1991),  the  present  investigator 
generally  started  by  attempting  to  make  a  three  category  discrimination:  current  level  IV, 
below  IV,  or  above  IV.  And  although  Stamp’s  level  I  through  level  VI  rating  of  each 
phrase  chosen  as  "most"  was  considered,  scoring  was  based  primarily  on  each 
interviewee’s  verbal  explanation  of  both  their  "most"  and  "least"  choices.  Only  if  there 
was  a  marked  discrepancy  between  the  estimates  obtained  from  analysis  of  the  Symbols 
and  Phrases  performances  were  notes  concerning  the  Work  History  consulted  in 
determining  the  overall  current  work  capacity  score. 

Subjects  and  Procedure 

Subjects  were  148  active  duty  Army  Officers  in  residence  at  the  US  Army  War 
College,  Carlisle  Barracks,  Pennsylvania.  Each  volunteered  to  be  interviewed  in 
connection  with  an  Army  "brown  envelope"  study  of  officer  development.  The  CPA 
interviews  took  approximately  90  minutes  and  were  conducted  by  individuals  familiar 
with  the  CPA  assessment  procedure.  Interviews  were  tape  recorded  and  the  tapes 
subsequently  transcribed.  One  hundred  and  thirty-seven  (137)  of  these  interviews  were 
scored  by  the  present  investigator  in  the  manner  described  above.  In  addition,  52  of  the 
137  were  scored  by  an  expert  :n  Stratified  Systems  Theory.  The  present  investigator  was 
kept  ignorant  of  those  scores  until  after  reliability  scoring  was  completed. 
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Task  One:  CPA  Reliability  Assessment 


Task  One  of  the  Statement  of  Work  called  for  the  determination  of  the  rater- 
rater  reliability  of  the  CPA  scores.  In  accomplishing  this  task,  the  present  investigator 
supervised  the  typing  of  78  Phrases  narratives  and  typed  several  others  himself.  The 
remaining  Phrases  typescripts  were  provided  to  the  present  investigator  by  the 
Contracting  Officer’s  Technical  Representative.  In  all,  137  complete  scoring  protocols 
were  obtained  and  scored  by  the  present  investigator.  As  shown  in  Table  1,  current 
conceptual  work  capacity  scores  (CLPL)  ranged  from  a  low  of  7  (low  current  level  III)  to 
a  high  of  15  (high  current  level  V).  The  majority  of  the  subjects  (58%)  obtained  scores 
in  the  level  IV  range  (scores  ranging  from  10  to  12)\  Twenty-six  percent  (26%)  were 
estimated  to  be  at  Jaques’  level  III,  and  four  percent  (4%)  at  level  V.  The  remainder 
were  scored  at  the  border  between  level  in  and  level  IV  (6%)  or  between  level  IV  and 
level  V  (6%). 


Table  1 

War  College  Students’  Current  Conceptual  Work  Capacity  Scores 


Cumulative  Cumulative 

CLPL  Frequency  Percent  Frequency  Percent 


7.0 

6 

4.4 

6 

4.4 

7.5 

2 

1.5 

8 

5.8 

8.0 

7 

5.1 

15 

10.9 

8.5 

5 

3.6 

20 

14.6 

9 

15 

10.9 

35 

25.5 

9.5 

8 

5.8 

43 

31.4 

10 

30 

21.9 

73 

53.3 

10.5 

13 

9.5 

86 

62.8 

11 

22 

16.1 

108 

78.8 

11.5 

5 

3.6 

113 

82.5 

12 

10 

7.3 

123 

89.8 

12.5 

8 

5.8 

131 

95.6 

13 

1 

0.7 

132 

96.4 

13.5 

4 

2.9 

136 

99.3 

15 

1 

0.7 

137 

100.0 

CLPL  =  Current  Work  Capacity  Scores  by  PL.  N  =  137 


’  Note  that  half  scores  were  assigned  in  those  instances  where  it  was  not 
possible  to  decide  between  two  adjacent  scores. 
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In  order  to  assess  rater-rater  reliability  of  the  CPA  current  capacity  scores, 
Pearson  product-moment  correlations  were  computed  between  CPA  scores  of  the  fifty- 
two  subjects  who  were  scored  by  both  the  current  investigator  (PL)  and  Jacobs  (TOJ). 
The  correlation  between  the  two  sets  of  current  work  capacity  scores  was  .81  (p<.0001, 
df=50).  For  several  reasons,  this  is  considered  to  be  an  excellent  level  of  rater-rater 
agreement.  First,  CPA  scores  result  from  a  very  complex  scoring  process  where  the 
scorer  makes  judgements  about  level  of  capability  based  on  both  the  form  and  substance 
of  the  subject’s  performance  on  a  number  of  taslu.  Second,  in  the  present  study  many  of 
Jacobs’  scores  were  assigned  immediately  after  he  conducted  the  CPA  interview  and  thus 
were  undoubtedly  influenced  by  both  the  Work  History  and  by  interpersonal  and 
nonverbal  cues  not  available  to  the  present  investigator.  Finally,  the  range  of  scores  was 
restricted  (from  low  III  to  high  V)  which  undoubtedly  attenuated  the  rater-rater 
agreement  correlation. 

Although  not  a  formal  part  of  Task  One  of  the  Statement  of  Work  for  the  present 
project,  additional  scores  became  available  on  a  subset  of  the  subjects,  scores  which  have 
a  bearing  upon  the  reliability  of  the  CPA  scores.  For  thirty-two  of  the  137  subjects 
assessed  with  the  CPA  by  the  present  investigator,  several  ratings  of  these  subjects  were 
obtained  from  two  sets  of  War  College  seminar  group  instructors.  Each  instructor 
provided  two  ratings  of  strategic  thinking  potential  and  a  rating  of  general  officer 
potential  for  the  students  in  his  seminar  group.  Because  current  conceptual  capacity  (as 
assessed  using  the  CPA)  is  expected  to  influence  students’  performance  in  the  War 
College’s  intensive,  issue  focused  seminar  groups,  instructor  rated  strategic  thinking 
capacity  and  general  officer  potential  can  be  considered  relevant  to  the  construct  validity 
of  conceptual  work  capacity. 

Each  of  the  32  subjects  (in  one  of  five  participating  seminar  groups)  was  rated 
independently  by  two  seminar  group  instructors.  Each  set  of  two  instructors  interacted 
with  the  students  in  the  same  seminar  group  but  at  different  points  in  the  academic  year; 
and  the  topical  focus  of  the  groups  was  different  under  the  different  sets  of  instructors. 
Thus,  the  two  sets  of  instructor  ratings  were  based  on  different  samples  of  the  students’ 
behavior.  Because  of  this  fact,  the  two  sets  of  instructor  ratings  were  expected  to  be  only 
moderately  correlated.  This  was  indeed  the  case.  The  two  sets  of  instructors’  paired 
comparison  ratings  (McAnulty,  1990)  of  strategic  thinking  potential  correlated  .57 
(p<.001,  df=30)  and  their  quartile  rankings  of  strategic  thinking  potential  correlated  .51 
(p<.01,  sif=30).  The  correlation  between  the  two  sets  of  instructors’  paired  comparison 
ratings  of  general  officer  potential  were  lower  (.34)  and  statistically  not  significantly 
different  from  zero  for  this  small  sample  (N=32). 

To  examine  the  construct  validity  associated  with  the  CPA  ratings  of  current 
conceptual  capacity,  the  correlations  among  current  level  scores  (CLPL)  and  a  sum  of 
the  two  instructors’  ratings  were  computed.  These  correlations  were  all  positive  and 
statistically  significant  beyond  the  .02  probability  level.  The  Pearson  correlations 
between  the  current  investigator’s  CPA  scores  and  the  combined  strategic  thinking  scores 
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were  .42  (paired  comparison  score),  .41  (quartile  score),  and  for  general  officer  potential 
was  .46.  These  correlations  not  only  provide  needed  construct  validity  support  for  the 
CPA  scores,  by  inference  they  also  support  the  reliability  of  the  CPA  scores  assigned  by 
the  current  investigator. 

Task  Two:  Improvement  of  CPA  Scoring  Procedures 

Analysis  of  PHRASES  Item  Sets.  Part  1  of  Task  Two  in  the  Statement  of  Work 
entails  conducting  an  analysis  of  the  PHRASES  item  sets  from  the  CPA  to  determine 
whether  their  scoring  and  overall  internal  consistency  can  be  improved.  At  issue  is  the 
extent  to  which  each  of  the  six  item  phrase  sets  (there  are  nine  sets  in  all)  can  be 
objectively  scored  and  combined  to  produce  an  overall  score  which  is  associated  with 
conceptu^  work  capacity.  In  each  item  set  (A  through  I)  there  are  six  phrases,  each  of 
which  is  thought  to  reflect  a  discrete  level  of  work  capacity.  Subjects  are  instructed  to 
select  the  one  phrase  which  is  "most"  like  the  way  they  prefer  to  approach  a  piece  of 
work  and  one  which  is  "least"  like  the  way  they  prefer  to  operate  at  work.  The  most 
parsimonious  way  to  arrive  at  an  objective  score  from  this  method  is  to  compute  an 
average  of  each  subject’s  "most"  choices^. 

Examination  of  the  relationship  between  each  subject’s  "most"  average  and  his  or 
her  rated  current  conceptual  capacity  (a  rater  judgement  based  on  a  subject’s  entire  CPA 
performance)  indicates  that  the  objective  index  ("most"  average)  is  indeed  correlated 
with  rated  current  capacity.  For  the  131  subjects  with  both  a  "most"  average  and  current 
capacity  score,  as  rated  by  PL,  the  Pearson  correlation  was  .59  (p<.0001,  df=129). 

For  49  subjects  similarly  rated  by  TOJ,  the  Pearson  correlation  was  .60  (p<.0001, 
df=47.  By  way  of  comparison,  the  correlation  between  subjects’  "least"  average  and 
rated  current  capacity  was  slightly  lower  (-.52  and  -.56  for  PL  and  TOJ  respectively). 
These  findings  suggest  that  attempts  to  improve  the  psychometric  properties  of  the 
PHRASES  item  sets  as  part  of  a  search  for  an  objective  scoring  method  for  the  CPA 
may  prove  fruitful. 

Table  2  shows  the  Cronbach  Coefficient  Alphas  computed  on  subjects’  "most" 
choices.  As  shown,  these  are  .776  for  the  raw  scores  and  .783  for  the  standardized  "most" 
scores.  The  alpha  statistic  is  an  estimate  of  the  internal  consistency  of  the  PHRASES 
"most"  responses  and  suggests  that,  as  currently  keyed,  PHRASES  has  an  acceptable 
level  of  internal  coherence.  A  more  fine  grained  analysis  of  the  contribution  of  each 
item  set  to  the  total  "most"  score  is  gained  by  examining  the  item-total  correlations  of 
each  item  set  and  the  effect  on  the  overall  Cronbach  Coefficient  Alpha  if  the 
contribution  of  each  item  set  is  deleted  (see  the  second  portion  of  Table  2). 


^  An  average  score  is  used  rather  than  a  total,  because  subjects  occasionally  fail 
to  indicate  an  item  preference. 
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Table  2 


Item  Analyses  for  Phrases  "Most"  Responses 

Cronbach  Coefficient  Alpha 

for  RAW  variables  :  0.775590 
for  STANDARDIZED  variables:  0.782659 


Raw  Variables  Std.  Variables 

Deleted  Correlation  Correlation 


Variable 

with  Total 

Alpha 

with  Total 

Alpha 

SETA 

0.498980 

0.748145 

0.500374 

0.757140 

SETB 

0.586142 

0.736643 

0.597894 

0.742883 

SETC 

0.437684 

0.758172 

0.440561 

0.765643 

SETD 

0.679003 

0.726257 

0.683128 

0.730016 

SETE 

0.470593 

0.752569 

0.467297 

0.761864 

SETF 

0.400464 

0.763884 

0.394625 

0.772051 

SETG 

0.359901 

0.767216 

0.363789 

0.776294 

SETH 

0.325447 

0.771561 

0J34752 

0.780246 

SET  I 

0.440638 

0.761551 

0.436561 

0.766205 

Examination  of  these  statistics  indicates  that  item  sets  D,  B,  A,  and  E  correlate 
most  closely  with  the  total  score,  while  item  sets  H,  G,  and  F  are  more  tenuously 
associated  with  the  "most"  total  score.  Nonetheless,  the  item-total  correlations  are  all 
positive  (ranging  from  .33  to  .68),  and  internal  consistency  of  the  entire  set  would  not  be 
improved  by  the  elimination  of  any  one  item  set.  These  analyses,  therefore,  do  not 
support  the  elimination  of  any  of  the  nine  PHRASES  item  sets. 

It  may,  nonetheless,  be  possible  to  improve  the  PHRASES  item  sets  by  a) 
rewriting  items  that  appear  to  contribute  little  to  discriminating  among  work  capacity 
levels  and/or  b)  re-keying  items  that  appear  to  be  endorsed  by  subjects  at  different 
capacity  levels  than  those  indicated  by  the  item’s  predetermined  level  score.  Such  an 
analysis  requires  identification  of  distinct  groups  of  subjects  who  have  been  determined 
to  be  functioning  at  different  levels  of  conceptual  work  capacity.  Because  of  the 
characteristics  of  the  present  sample  of  War  College  students  (see  Table  1),  it  was 
decided  to  divide  subjects  into  four  groups  based  on  PL’s  current  level  scores:  Current 
Level  III  (scores  up  to  and  including  9.0),  low  Current  Level  IV  (scores  from  9.5  to  10.5), 
high  Current  Level  FV  (scores  from  11.0  to  12.0),  and  Current  Level  V  (scores  12.5  and 
above).  Included  were  96  subjects  who  met  the  following  criteria:  1)  they  received  a 
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current  level  score  from  PL,  2)  their  PHRASES  "most"  responses  were  available  for  at 
least  8  of  the  9  item  sets,  3)  there  was  no  group  membership  discrepancy  between 
current  level  scores  assigned  by  PL  and  TOJ,  and  4)  there  was  no  group  membership 
discrepancy  of  more  than  1/2  point  between  rated  overall  current  conceptual  level  and 
current  level  based  only  on  the  scoring  of  the  subject’s  PHRASES  responses. 

Table  3  (Appendix  A)  shows  the  distribution  of  "most"  responses  to  the  items  in 
Set  A  for  subjects  classified  into  the  four  current  level  groups  described  above.  This 
item  set,  which  had  an  item-total  correlation  of  .50,  seemed  reasonably  well  constructed 
with  all  items  receiving  some  endorsements  and  higher  level  subjects  generally  endorsing 
the  higher  keyed  items. 

Table  4  (Appendix  A)  shows  a  similar  analysis  of  item  set  B.  This  set  had  one  of 
the  highest  item-total  correlations  (.59)  and  also  appeared  to  be  well  constructed  and 
keyed.  Table  5  (Appendix  A)  shows  the  distribution  of  "most"  responses  to  Set  C.  This 
set  had  a  more  modest  correlation  with  the  "most"  total  (.44,  with  its  own  contribution 
removed).  Examination  of  the  fi’equency  distributions  reveals  that  this  item  set’s  ability 
to  discriminate  among  the  four  current  level  groups  is  hampered  by  a  high  level  of 
endorsement  (>66%)  of  item  5,  "See  the  rules  as  guides  to  action",  by  subjects  in  all  but 
the  highest  group.  In  an  attempt  to  make  other  phrases  in  this  item  set  more  attractive, 
the  phrases  rated  III  and  FV  by  Stamp  (1984)  were  rewritten.  The  revised  item  set  can 
be  found  in  Appendix  B. 

Table  6  (Appendix  A)  shows  the  distribution  of  "most"  responses  among  the  items 
of  Set  D.  This  set  had  the  highest  item-total  correlation  (.68)  of  all  the  PHRASES  sets. 
Interestingly,  not  a  single  subject  endorsed  item  5,  "Restructure  the  task.”  Although 
there  is  a  risk  that  replacing  this  low  endorsement  item  might  reduce  the  item  set’s 
correlation  with  the  total  score,  a  more  specific  item  might  attract  more  subjects.  In 
consultation  with  the  P.O.C.,  it  was  decided  to  replace  item  5  with  the  following  item: 
"Look  for  relationships  between  the  current  task  and  other  tasks,"  The  revised  item  set 
can  be  foimd  in  Appendix  B. 

Responses  to  the  items  in  Set  E  are  presented  in  Table  7  (Appendix  A).  This  set 
had  a  reasonably  good  item-total  correlation  (.47),  but  item  3,  "Look  for  sequences  or 
common  re’-^tionships,"  had  a  very  low  frequency  of  endorsement.  A  possible  rewrite  of 
this  item  is  Figure  out  the  right  sequence  of  tasks."  The  revised  item  set  can  be  found 
in  Appendix  B. 

Set  F  had  one  of  the  lowest  item-total  correlations  (.40)  of  the  PHRASES  item 
sets.  Inspection  of  Table  8  (Appendix  A)  indicates  the  main  reason  for  this  was  a  very 
high  endorsement  frequency  of  item  1,  "Use  your  common  sense,"  by  all  but  the  subjects 
in  the  current  level  V  group.  It  would  seem  advisable  to  eliminate  this  item  from  the 
item  set,  at  least  for  an  Army  officer  population,  where  common  sense  appears  to  be  a 
highly  valued  commodity.  A  revision  of  this  item  set  which  replaces  the  above  item  and 
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revises  two  other  items  can  be  found  in  i^pendix  B. 


Set  G  is  another  item  set  with  a  relatively  poor  item-total  correlation  (r=.36). 
Inspection  of  Table  9  (i^pendix  A)  suggests  that  the  major  problem  with  this  item  set  is 
that  a  relatively  high  proportion  of  current  level  in  and  Lo  IV  subjects  endorsed  item  6, 
"See  gaps  as  pauses  in  the  process."  This  item  was  rewritten  as  "Know  that  new 
information  creates  new  gaps."  and  the  original  item  substituted  for  "See  gaps  as  missing 
links  in  a  chain,"  which  had  very  low  endorsement  frequencies  (6  of  95  endorsements). 
The  revised  item  set  can  be  found  in  i^pendix  B. 

Set  H  had  the  lowest  item-total  correlation  (33).  Inspection  of  Table  10 
(Appendix  A)  reveals  no  obvious  reason  why  this  should  be  the  case.  One  minor 
problem  appears  to  be  that  the  keying  of  items  3  and  4  may  be  reversed.  Another 
possibility  is  that  item  S,  "Seek  original  solutions,"  may  have  a  high  level  of  social 
desirability.  In  consultation  with  the  P.O.C.,  several  items  in  this  set  were  revised  or 
rewritten  to  try  to  boost  its  validity.  The  revised  item  set  can  be  found  in  Appendix  B. 

Finally,  Table  11  (Appendix  A)  shows  the  "most"  endorsements  for  item  set  I. 

The  only  obvious  problem  with  this  item  set  is  that  there  is  a  high  level  of  endorsement 
of  item  1,  "Answers  should  be  straightforward."  This  investigator’s  impression  from 
reading  a  large  number  of  subjects’  justifications  of  this  item  endorsement  is  that  many 
current  level  IV  subjects  were  careful  to  point  out  that  even  when  it  is  desirable  to  give 
straightforward  answers,  the  problems  may  be  quite  complicated.  It  might  be  possible  to 
rewrite  this  item  as  follows  to  reduce  the  number  of  current  level  IVs  who  endorse  it: 
"Most  problem  solutions  are  straightforward."  The  resulting  new  item  set  can  be  found 
in  Appendix  B. 

To  summarize,  the  PHRASES  item  sets  had  reasonable  internal  consistency,  as 
reflected  by  Cronbach’s  Coefficient  Alpha.  And  while  an  examination  of  item  responses 
broken  down  by  subject  level  suggested  a  number  of  ways  in  which  some  of  the  item  sets 
might  be  improved,  there  was  little  evidence  that  eliminating  one  or  more  item  sets 
would  appreciably  improve  internal  consistent^  of  the  entire  PHRASES  task. 

Analysis  of  SYMBOLS  Statistics.  The  second  part  of  Task  Two  of  the  Statement 
of  Work  was  an  exploration  of  the  best  way  to  use  the  SYMBOLS  data  to  arrive  at  an 
estimate  of  subjects’  Current  Cone-  p'ual  Work  Capacity.  More  specifically,  were 
statistics  other  than  total  number  of  cards  to  criterion,  either  separately  or  in 
combination  with  the  total,  better  indicators  of  a  subject’s  current  level  than  total  cards 
to  criterion? 

To  address  this  research  question  a  new  data  set  was  created  which  included  only 
those  subjects  who  had  a  computer  generated  Symbols  performance,  whose  performance 
was  not  "aborted,"  and  who  received  a  current  conceptual  level  score  from  the  present 
investigator  (PL).  One  hundred  and  fifteen  (115)  of  the  entire  sample  of  148  met  these 


criteria  and  were  subjected  to  the  regression  analyses  described  below. 


Table  12 

Multiple  Reeression  of  SYMBOLS  Data  on  Current  Conceptual  Level  (CLPU 
Maximum  R-square  Improvement  for  Dependent  Variable  CLPL 
Step  1  Variable  MOVES  Entered  R-square  =  03350  C(p)  =  2.1036 


DF 

Sum  of 

Mean 

F 

Prob>F 

Squares 

Square 

Regression 

1 

1.93534 

81.93534 

56.92 

0.0001 

Error 

113 

162.66032 

1.43947 

Total 

114 

244.59565 

Step 

2  Variable  PROPCRCT  Entered 

R-square 

=  03461 

DF 

Sum  of 

Mean 

F 

Prob>F 

Squares 

Square 

Regression 

2 

84.83419 

42.41710 

29.74 

0.0001 

Error 

112 

159.76146 

1.42644 

Total 

114 

244.59565 

=  2.0879 


Step  3  Variable  TIME  Entered  R-square  =  0.3474  C(p)  =  4.0000 


DF 

Sum  of 

Mean 

F 

Prob>F 

Squares 

Square 

Regression 

3 

84.96059 

28.32020 

19.69 

0.0001 

Error 

111 

159.63507 

1.43815 

Total 

114 

244.59565 

No  further  improvement  in  R-square  is  possible. 


A  Maximum  R-square  stepvrise  multiple  regression  procedure  (SAS  release  6.04) 
was  used  to  determine  which  of  the  SYMBOLS  statistics  were  independently  and  in 
combination  the  most  highly  correlated  with  investigator  rated  Current  Conceptual  Level. 
The  Maximum  R-square  procedure  is  used  to  find  the  "best"  n-variable  regression  models 
starting  with  the  best  sin^e  variable  model.  Next,  the  best  two  variable  model  is 
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identified,  the  one  which  yields  the  greatest  increase  in  variance  over  the  single  variable 
model.  This  procedure  continues  until  no  appreciable  increase  in  variance  is  obtained 
when  additional  variables  are  included  or  there  are  no  more  predictor  variables  to  be 
added.  The  three  predictor  variables  were  Phrases  Most  Average  (MOST),  Phrases 
Least  Average  (LEAST),  Symbols  Moves  Total  (MOVES),  Proportion  of  Correct 
Symbols  Moves  (PROPCRCT),  and  Average  Time  of  Symbols  Moves  (TIME). 

In  the  first  regression  analysis  three  statistics  derived  from  the  computer 
generated  SYMBOLS  task  were  treated  as  potential  predictors  of  Current  Conceptual 
Capacity,  as  rated  by  the  present  investigator  (PL).  These  statistics  were  total  number  of 
cards  to  criterion  (or  to  the  end  of  the  task,  162  cards),  average  time  taken  for  each  card 
sort,  and  the  proportion  of  total  cards  sorted  which  were  correctly  sorted.  The  results  of 
this  analysis  can  be  found  in  Table  12.  Step  1  selected  total  number  of  moves  as  the 
best  single  variable  predictor  of  investigator  rated  Current  Conceptual  Level.  This 
variable  produced  an  R^  of  .335  (p<.0001,  df=  !)•  In  step  2,  the  total  number  of  cards 
and  proportion  of  total  cards  which  were  correctly  sorted  were  selected  for  the  best  two 
variable  model.  Inclusion  of  these  two  variable  in  the  predictor  equation  resulted  in  a 
small  and  statistically  nonsignificant  increase  in  R*  from  .335  to  .347.  The  effect  of  the 
addition  of  the  third  predictor  to  the  equation,  average  time  per  move,  also  produced  a 
negligible  increase  in  R^  To  summarize  this  analysis,  it  appears  that  only  total  number 
of  moves  in  the  SYMBOLS  task  is  a  significant  predictor  of  rated  conceptual  level. 

The  same  Maximum  R-square  analysis  described  immediately  above  was  also 
conducted  for  the  36  subjects  in  the  data  set  who  received  a  Current  Conceptual  Level 
score  fi’om  T.  O.  Jacobs.  With  such  a  small  sample  size,  these  results  should  be  treated 
with  great  caution.  The  results  of  the  analysis  of  SYMBOLS  predictors  of  Jacobs’ 
Current  Conceptual  Level  scores  are  shown  in  Table  13.  As  can  be  seen,  none  of  the 
predictors  accounted  for  an  R^  significantly  different  from  zero  for  this  small  sample. 
Total  number  of  moves  was  the  best  predictor  of  Jacobs’  scores,  but  the  R^  was  only 
.052.  It  seems  likely  that  Jacobs  relied  less  on  SYMBOLS  performance  than  did  the 
present  investigator  in  assigning  an  overall  Current  Conceptual  Level  score. 

A  final  question  which  can  be  addressed  using  the  present  data  set  concerns  the 
extent  to  which  PHRASES  statistics  ("Most"  and  "Least"  averages)  can  be  combined  with 
SYMBOLS  statistics  to  improve  prediction  of  rated  Conceptual  Level  over  and  above  the 
use  of  either  set  of  statistics  alone.  To  examine  this  question,  an  additional  Maximum 
R-square  regression  analysis  was  conducted  where  both  PHRASES  and  SYMBOLS 
statistics  were  included  as  possible  predictors  of  Current  Conceptual  Level,  as  rated  by 
the  current  investigator  (PL).  The  results  of  this  analysis  are  presented  in  Table  14. 

The  results  were  dramatic  and  rather  straightforward.  As  expected  fi'om  an  examination 
of  the  first  order  correlations,  the  single  best  predictor  of  Current  Conceptual  Level  was 
"Most"  average  firom  the  PHRASES  task.  This  variable  produced  an  R^  of  .374 
(p<.0001,  df=l)-  The  procedure  next  selected  Symbols  total  cards  as  contributing  the 
most  to  a  two  variable  prediction  equation,  and  the  increase  in  R^  was  dramatic.  The  R^ 
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Table  13 


Multiple  Regression  of  SYMBOLS  Data  on  Current  Conceptual  Level  (CLTOJ) 
Maximum  R-square  Improvement  for  Dependent  Variable  CLTOJ 

Step  1  Variable  MOVES  Entered  R-square  =  0.0516  C(p)  =  0.1775 


DF 

Sum  of 

Mean 

F 

Prob>F 

Squares 

Square 

Regression 

1 

5.85627 

5.85627 

1.85 

0.1826 

Error 

34 

107.55345 

3.16334 

Total 

35 

113.40972 

Step  2  Variable  TIME 

Entered  R-square  = 

0.0569  C(p) 

DF 

Sum  of 

Mean  F 

Prob>F 

Squares 

Square 

Regression  2 

6.44830 

3.22415  0.99 

0.3806 

Error  33 

106.96142 

3.24126 

Total  35 

113.40972 

Step  3  Variable  PROPCRCT  Entered  R-square  =  0.0569  C(p)  =  4.0000 


DF 

Sum  of 

Mean 

F 

Prob>F 

Squares 

Square 

Regression 

3 

6.44957 

2.14986 

0.64 

0.5929 

Error 

32 

106.96016 

3.34250 

Total 

35 

113.40972 

The  above  model  is  the  best  3-variable  model  found.  No  further  improvement  in 
R-square  is  possible. 
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Table  14 


Multiple  Re£ression  of  PHRASES  and  SYMBOLS  Data  on  Current  Conceptual  Uvel  fCLPU 


Maximum  R-square  Improvement  for  Dependent  Variable  CLPL 


Step  1  Variable  MOST  Entered  R-square  =  0.3739  C(p)  =  58.0320 


DF 

Sum  of 

Mean 

F 

Prob  >  F 

Squares 

Square 

Regression 

1 

89.44958 

89.44958 

65.69 

0.0001 

Error 

110 

149.78033 

136164 

Total 

111 

239.22991 

Step  2  Variable  MOVES  Entered  R-square  =  0.5754  C(p)  =  6.5947 


DF 

Sum  of 

Mean 

F 

Prob>F 

Squares 

Square 

Regression 

2 

137.65630 

68.82815 

73.86 

0.0001 

Error 

109 

101.57362 

0.93187 

Total 

111 

239.22991 

Step  3  Variable  PROPCRCT  Entered  R-square  =  0.5935  C(p)  =  3.7925 


DF 

Sum  of 

Mean 

F 

Prob>F 

Squares 

Square 

Regression 

3 

141.98839 

47.32946 

52.57 

0.0001 

Error 

108 

97.24152 

0.90038 

Total 

111 

239.22991 

Step  4  Variable  LEAST  Entered  R-square  =  0.5973  C(p)  =  4.8006 


DF 

Sum  of 

Mean 

F 

Prob>F 

Squares 

Square 

Regression 

4 

142.88322 

35.72080 

39.67 

0.0001 

Error 

107 

96.34669 

0.90044 

Total 

111 

239.22991 

Step  5  Variable  TIME  Entered  R-square  =  0.6003  C(p)  =  6.0000 


DF 

Sum  of 

Mean 

F 

Prob>F 

Squares 

Square 

Regression 

5 

143.60546 

28.72109 

31.84 

0.0001 

Error 

106 

95.62445 

0.90212 

Total 

111 

239.22991 
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increased  from  .374  to  .575,  and  this  increase  was  statistically  highly  significant  (i2<.0001, 
S^=2).  Together,  these  two  predictors  produce  a  multiple  r  of  .76,  which  approachesthe 
rated  reliability  of  both  the  CPA  score  (.81)  and  the  PHRASES  "Most"  score  (Coefficient 
Alpha  =  .78).  The  third  variable  selected  by  the  Maximum  R-square  procedure,  the 
Symbols  proportion  correct  statistic,  also  produced  a  small  but  statistically  significant 
increase  in  the  multiple  R^  from  .575  to  .594  (p<.05,  df=3). 


SUMMARY,  CONCLUSIONS,  AND  RECOMMENDATIONS 

In  general,  the  present  investigation  of  the  Career  Path  Appreciation  (CPA) 
assessment  method  yielded  results  which  support  the  value  of  conducting  further 
investigations  of  this  approach.  First,  the  CPA  Current  Conceptual  Level  scores  appear 
to  have  adequate  rater-rater  reliability.  Despite  the  fact  that  scoring  the  CPA  entails 
making  complex  judgements  about  subjects’  multiple  responses,  and  that  the  two  scorers 
often  consulted  different  parts  of  the  CPA  data  in  making  scoring  judgements,  the 
Pearson  correlation  between  the  two  raters’  scores  was  a  respectable  .81.  This  rater- 
rater  reliability  index  is  the  first  known  reliability  estimate  for  the  CPA.  In  addition, 
modest  construct  validity  was  demonstrated  for  the  CPA  Current  Conceptual  Capacity 
scores  provided  by  the  present  investigator.  War  college  instructors,  who  rated  a 
subsample  of  our  subjects’  strategic  thinking  skill  as  demonstrated  in  War  College 
seminar  groups,  tended  to  rate  those  who  scored  higher  on  the  CPA  as  being  better 
thinkers  than  those  who  scored  lower  on  the  CPA  (r  =  .57  and  .51  on  two  different 
thinking  skill  measures).  Given  the  dependence  of  validity  estimates  on  reliability,  this 
validity  finding  lends  further  support  to  the  reliability  of  the  CPA  scores. 

Search  for  possible  objective  approaches  to  scoring  the  CPA  also  yielded 
promising  results.  Analysis  of  the  Phrases  portion  of  the  CPA  yielded  respectable 
Cronbach  Coefficient  Alphas  for  the  Phrases  "Most"  scores  and  these  scores  were 
reasonably  good  predictors  of  rated  Current  Conceptual  Capacity  scores.  Furthermore, 
item  analyses  of  the  Phrases  item  sets  suggested  that  all  nine  item  sets  should  be 
retained.  Similar  analyses  of  the  objective  indices  generated  by  the  computer  controlled 
Symbols  task  of  the  CPA  showed  that  the  index  of  the  total  number  of  cards  sorted  is 
also  a  reasonably  good  predictor  of  rated  Current  Conceptual  Capacity,  at  least  for  some 
raters.  Most  encouraging  of  all,  in  terms  of  the  likelihood  of  being  able  to  develop  an 
objective  scoring  procedure  for  the  CPA,  is  the  finding  that  a  combination  of  the 
principal  objective  scores  from  the  Phrases  and  Symbols  tasks  (Phrases  "Most"  average 
and  Symbols  total  cards  sorted),  when  combined,  produce  a  level  of  prediction  of  rated 
CPA  scores  which  is  very  close  to  the  reliability  estimates  of  the  measures.  Finally, 
examination  of  individual  responses  to  the  Phrases  task  by  subjects  classified  into  four 
conceptual  capacity  levels  on  the  basis  of  their  investigator  rated  CPA  performances 
suggested  ways  in  which  some  of  the  Phrases  item  sets  might  be  improved. 

Overall,  the  present  investigation  suggests  that  the  CPA  assessment  methodology 
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may  be  a  good  candidate  for  the  assessment  of  individual  conceptual  characteristics 
which  are  believed  to  be  essential  to  the  effective  accomplishment  of  complex 
organizational  work.  It  should  be  kept  in  mind,  however,  that  the  present  sample  of  U.S. 
Army  War  College  students  is  a  highly  homogeneous  group  with  respect  to  age,  gender, 
organizational  level,  and  work  sociali^tion  background.  Therefore,  generalization  of  the 
present  findings  to  other  populations  would  appear  to  be  unwarranted.  Nonetheless,  the 
strength  of  the  present  results  are  testimony  to  the  power  of  Stratified  Systems  Theory, 
from  which  the  CPA  was  derived,  and  to  the  work  of  Gillian  Stamp  in  developing  the 
CPA  assessment  approach. 
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APPENDIX  A 


Bar  Graphs  Showing  Frequency  of  Selection  of  CPA  PHRASES  Items 
For  the  Nine  Item  Sets  Broken  Down  by  Subject  Group 
For  96  War  College  Students 


A-1 


Table  3 


"Most"  Responses  to  Set  A  of  CPA  PHRASES  bv  Subjects  Classified 
at  Four  Different  Current  Capacity  Levels  (III.  LoIV.  HilV.  and 

Yi 

LEVEL=III 


FREQUENCY  OF  SET  A 


CUM 

1 

FREQ 

FREQ  PERCENT 

1 

j  ******** 

4 

4 

12.90 

2 

************************************  18 

22 

58.06 

3 

** 

1 

23 

3.23 

4 

** 

1 

24 

3.23 

5 

4 

28 

12.90 

6 

I  "kitifkitit 

3 

31 

9.68 

- J. — 

-+ 

2 

4  6  8  10  12  14  16 

18 

FREQUENCY 

LEVEL=LoIV 

FREQUENCY  OF  SET  A 

CUM 

1 

FREQ 

FREQ 

PERCENT 

1 

**  1 

1 

3.57 

2 

************************  12 

13 

42.86 

3 

******  3 

16 

10.71 

4 

**  1 

17 

3.57 

5 

**********  5 

22 

17.86 

6 

**********  5 

27 

17.86 

7  1 

**  1 

28 

3.57 

2  4  6  8  10  12 

FREQUENCY 


Phrases  - 

1  -  Work  to  a  complete  set  of  instructions 

2  -  Work  within  a  given  framework 

3  -  Work  with  connections  even  if  particular  links  are 

unclear 

4  -  Work  in  abstracts  and  concepts 

5  -  Work  with  a  minimum  of  preconceptions 

6  -  Define  the  horizons  of  the  work 

7  -  (Two  or  more  phrases  identified  as  "Most") 


A-2 


Table  3  Continued 


LEVEIj=HiIV 


FREQUENCY  OF  SET  A 


CUM 


FREQ 

FREQ 

PERCENT 

1 

1 

j  -klfkltlfitltlililt 

2 

2 

8.33 

2 

j  ****itit*ikA****Ait 

3 

5 

12.50 

3 

********** 

2 

7 

8.33 

4 

*************** 

3 

10 

12.50 

5 

j  ****************************** 

6 

16 

25.00 

6 

*********************************** 

7 

23 

29.17 

7 

***** 

1 

24 

4.17 

.+ - + - + - + - + - + - + 

1  2  3  4  5  6  7 

FREQUENCY 


LEVEL=V 


FREQUENCY  OF  SET  A 


SET  A 

CUM 

1 

FREQ 

FREQ 

PERCENT 

3 

********** 

2 

2 

15.38 

4 

j  ********** 

2 

4 

15.38 

5 

j  ***** 

1 

5 

7.69 

6 

j  ****************************** 

6 

11 

46.15 

7 

********** 

2 

13 

15.38 

.+ - + - + - + - + - + 

1  2  3  4  5  6 

FREQUENCY 
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Table  4 


"Most**  Responses  to  Set  B  of  CPA  PHRASES  by  Subjects  Classified 
at  Four  Different  Current  Capacity  Levels  (III.  Loiv.  Hiiv.  and 

VI 


LEVEL-III 


FREQUENCY  OF  SET  B 


1 

FREQ 

CUM 

FREQ 

PERCENT 

CUM 

PERCENT 

1 

j  hhlfkltlfkltlfkltlfk-k 

7 

7 

22.58 

22.58 

2 

******** 

4 

11 

12.90 

35.48 

3 

********************** 

11 

22 

35.48 

70.97 

4 

j  ************ 

6 

28 

19.35 

90.32 

5 

**** 

2 

30 

6.45 

96.77 

6 

** 

1 

31 

3.23 

100.00 

2  4  6  8  10 

FREQUENCY 


LEVEL=LoIV 


FREQUENCY  OF  SET  B 


CUM 

CUM 

1 

FREQ 

FREQ 

PERCENT 

PERCENT 

2 

j  **** 

2 

2 

7.14 

7.14 

3 

****************** 

9 

11 

32.14 

39.29 

4 

j  ************************ 

12 

23 

42.86 

82.14 

6 

**** 

2 

25 

7.14 

89.29 

7 

****** 

3 

28 

10.71 

100.00 

2  4  6  8  10  12 

FREQUENCY 


Phrases  - 

1  -  Do  one  thing  at  a  time 

2  -  Focus  on  one  part  of  the  task  at  a  time 

3  -  Co-ordinate  by  drawing  together  a  number  of  separate 

strands 

4  -  Compare  the  merits  of  alternative  options 

5  -  Establish  new  relationships  between  previously  unrelated 

materials 

6  -  Use  words,  ideas,  and  theories  as  tools 

7  -  (Two  or  more  phrases  identified  as  "Most”) 


A-4 


T2U3le  4  Continued 
LEVEL-HilV 


FREQUENCY  OF  SET  B 


CUM 

CUM 

FREQ 

1 

FREQ 

PERCENT 

PERCENT 

2 

********** 

2 

2 

8.33 

8.33 

3 

******************** 

4 

6 

16.67 

25.00 

4 

************************************ 

7 

13 

29.17 

54.17 

5 

j  *************** 

3 

16 

12.50 

66.67 

6 

******************** 

4 

20 

16.67 

83.33 

7 

j  ******************** 

4 

24 

16.67 

100.00 

■+ - + - + - + - + - + - + 

1  2  3  4  5  6  7 

FREQUENCY 


LEVEL=V 

FREQUENCY  OF  SET  B 


CUM 

CUM 

FREQ 

FREQ 

PERCNT 

PERCNT 

3 

1 

j  **** 

1 

1 

7.69 

7.69 

4 

**** 

1 

2 

7.69 

15.38 

5 

************************************  9 

11 

69.23 

84.62 

6 

1 

12 

7.69 

92.31 

7 

1 

13 

7.69 

100.00 

.+ - + - + - + - + - + - + - + - + 


123456789 

FREQUENCY 


Table  5 


a£_L 


Y1 


LEVE1>III 


FREQUENCY  OF  SET  C 


CUM 

CUM 

1 

FREQ 

FREQ 

PERCENT 

PERCENT 

1 

j  *** 

3 

3 

9.68 

9.68 

2 

***** 

5 

8 

16.13 

25.81 

4 

* 

1 

9 

3.23 

29.03 

5 

******************* 

19 

28 

61.29 

90.32 

6 

** 

2 

30 

6.45 

96.77 

7 

* 

1 

31 

3.23 

100.00 

— +-+-+-+“+-+-+-+-+- 
2  6  10  14  18 

FREQUENCY 


LEVEL=LoIV 


FREQUENCY  OF  SET  C 


1 

FREQ 

CUM 

FREQ 

PERCENT 

CUM 

PERCENT 

1 

!  * 

1 

1 

3.57 

3.57 

3 

* 

1 

2 

3.57 

7.14 

5 

j  it  a  it  it  it  it  it  it  it  iiit  it  it  Hr  it  it  if  it  it  It  it  21 

23 

75.00 

82.14 

6 

* 

1 

24 

3.57 

85.71 

7 

**** 

4 

28 

14.29 

100.00 

— +-+-+-+-+-+-+-+-+-+- 
2  6  10  14  18 

FREQUENCY 


Phrases  - 

1  -  Follow  the  rules 

2  -  Work  within  the  rules 

3  -  Extrapolate  from  the  given  rules 

4  -  Look  for  the  pattern  of  the  rules 

5  -  See  the  rules  as  guides  to  action 

6  -  Redefine  the  rules 


7  -  (Two  or  more  phrases  identified  as  "Most") 
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Table  5  Continued 


LEVEL^HilV 


FREQUENCY  OF  SET  C 


CUM 

FREQ  FREQ  PERCENT 


3 

4 

5 
7 


****  2 

****  2 

********************************  16 
********  4 

- + - + - + - + - + - + - + - + 

2  4  6  8  10  12  14  16 

FREQUENCY 


2 

4 

20 

24 


8.33 

8.33 

66.67 

16.67 


LEVEL=V 


FREQUENCY  OF  SET  C 


3 

4 

5 

6 
7 


CUM 

FREQ  FREQ 

I 


CUM 

PERCENT  PERCENT 


*****  1 

j  **********  2 

********************  4 

********************  4 

j  **********  2 

- + - + - + - + 

12  3  4 

FREQUENCY 


1  7.69  7.69 

3  15.38  23.08 

7  30.77  53.85 

11  30.77  84.62 

13  15.38  100.00 


A-7 


CUM 

PERCENT 

8.33 

16.67 

83.33 

100.00 


Table  6 


"Most"  Responses  to  Set  D  of  CPA  PHRASES  bv  Subjects  Classified 
at  Four  Different  Current  Capacity  Levels  (III.  LoIV.  Hiiv.  and 

VI 


LEVEL=III 


FREQUENCY  OF  SET  D 


CUM 

CUM 

1 

FREQ 

FREQ 

PERCENT 

PERCENT 

1 

j  ******** 

4 

4 

12.90 

12.90 

2 

******** 

4 

8 

12.90 

25.81 

3 

jUtititititikititititiUtititititiritltikiltikitiUtititikitititit 

16 

24 

51.61 

77.42 

4 

************ 

6 

30 

19.35 

96.77 

6 

j  ** 

1 

31 

3.23 

100.00 

- + — + — + — + — + — 

-+ — 

_+ — 

-+ 

2  4  6  8  10 

12 

14 

16 

FREQUENCY 

IiEVEL=LoIV 


FREQUENCY  OP 

SET 

D 

CUM 

CUM 

FREQ 

1 

FREQ 

PERCENT 

PERCENT 

1 

j  ** 

1 

1 

3.57 

3. 

2 

**** 

2 

3 

7.14 

10.71 

3 

j  ************************ 

12 

15 

42.86 

53.57 

4 

******************** 

10 

25 

35.71 

89.29 

7 

j  ****** 

2  4  6  8  10  12 

FREQUENCY 

3 

28 

10.71 

100.00 

Phrases  - 


1  -  Follow  instructions  carefully 

2  -  Approach  each  task  in  own  right 

3  -  Take  a  systematic  approach 

4  -  Span  a  broad  spectrum  and  also  focus  in  detail 

on  certain  aspects 

5  -  Restructure  the  task 

6  -  Transcend  the  task 

7  -  (Two  or  more  phrases  identified  as  "Most”) 


A-8 


H  fg  n 


Table  6  continued 


LEVEL=HiIV 


FREQUENCY  OF  SET  D 


FREQ 

CUM 

FREQ 

PERCENT 

CUM 

PERCENT 

** 

1 

1 

4.17 

4.17 

** 

1 

2 

4.17 

8.33 

******************** 

10 

12 

41.67 

50.00 

******************** 

10 

22 

41.67 

91.67 

**** 

2 

24 

8.33 

100.00 

.+ + + + + 

2  4  6  8  10 

FREQUENCY 


LEVEL=V 


FREQUENCY  OF  SET  D 


3 

4 
6 
7 


FREQ 


****  2 

******  3 

j  ************  g 

j  ****  2 

2  4  6 


FREQUENCY 


CUM 

FREQ  PERCENT 

2  15.38 

5  23.08 

11  46.15 

13  15.38 


CUM 

PERCENT 

15.38 

38.46 

84.62 

100.00 


A-9 


Table  7 


"Moat"  Responses  to  Set  E  of  CPA  PHRASES  bv  Subjects  Classified 
at  Four  Different  Current  Capacity  Levels  (III.  LoIV^  Hiiv.  Vi 


LEVEL»III 


FREQUENCY  OF  SET  E 


1 

FREQ 

CUM 

FREQ 

PERCENT 

CUM 

PERCENT 

1 

j  ********************** 

11 

11 

35.48 

35.48 

2 

************ 

6 

17 

19.35 

54.84 

3 

** 

1 

18 

3.23 

58.06 

4 

************ 

6 

24 

19.35 

77.42 

5 

j  ********** 

5 

29 

16.13 

93.55 

6 

** 

1 

30 

3.23 

96.77 

7 

** 

1 

31 

3.23 

100.00 

- + - + - + - + - +- 

2  4  6  8  10 

FREQUENCY 

LEVEL=LoIV 


FREQUENCY  OF  SET  E 


1 

FREQ 

CUM 

FREQ 

PERCENT 

CUM 

PERCENT 

1 

j  ******** 

4 

4 

14.29 

14.29 

2 

j  ******** 

4 

8 

14.29 

28.57 

3 

** 

1 

9 

3.57 

32.14 

4 

******** 

4 

13 

14.29 

46.43 

5 

************************ 

12 

25 

42.86 

89.29 

6 

j  **** 

2 

27 

7.14 

96.43 

7 

** 

1 

28 

3.57 

100.00 

.+ - + - + - + - + - + 

2  4  6  8  10  12 

FREQUENCY 


Phrases  - 

1  -  Do  first  things  first 

2  -  Break  up  aproblem  into  separate  parts 

3  -  Look  for  sequences  or  common  relationships 

4  -*  Analyse  problems  by  searching  for  their  underlying 
structure 

-  Create  an  overall  picture  of  the  problem 

-  Consider  the  context  of  the  problem 

-  (Two  or  more  phrases  identified  as  "Most") 


5 

6 
7 


A-10 


w  n  ^  in  »<) 


TeQsle  7  Continued 


LEVEL-HilV 


FREQUENCY  OF  SET  E 


FREQ 

CUM 

FREQ 

PERCENT 

CUM 

PERCENT 

**** 

2 

2 

8.33 

8.33 

hh** 

2 

4 

8.33 

16.67 

******* 

4 

8 

16.67 

33.33 

****************** 

9 

17 

37.50 

70.83 

******** 

4 

21 

16.67 

87.50 

****** 

3 

24 

12.50 

100.00 

+ - + - + - + 

2  4  6  8 

FREQUENCY 


LEVEL«V 

FREQUENCY  OF  SET  E 


CUM  CUM 

FREQ  FREQ  PERCENT  PERCENT 

I 


3 

***** 

1 

1 

7.69 

7.69 

4 

******************** 

4 

5 

30.77 

38.46 

5 

************************* 

5 

10 

38.46 

76.92 

6 

********** 

2 

12 

15.38 

92.31 

7 

***** 

1 

13 

7.69 

100.00 

,+ - + - + - + - + 

1  2  3  4  5 

FREQUENCY 
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Table  8 


"Most"  Responses  to  Set  F  of  CPA  PHRASES  bv  Subjects  Classified 
at  Four  Different  Current  Capacity  Levels  fill.  Loiv.  Hiiv.  and 

LEVEL*III 


FREQUENCY  OF  SET  F 


1 

FREQ 

CUM 

FREQ 

PERCENT 

CUM 

PERCENT 

1 

******  4t  *******  it  *  it  ***********  * 

29 

29 

93.55 

93.55 

3 

* 

1 

30 

3.23 

96.77 

4 

* 

1 

31 

3.23 

100.00 

5  10  15  20  25 

FREQUENCY 


LEVEL=LoIV 


FREQUENCY  OF  SET  F 


1 

FREQ 

CUM 

FREQ 

PERCENT 

CUM 

PERCENT 

1 

j  ********************* 

21 

21 

75.00 

75.00 

3 

** 

2 

23 

7.14 

82.14 

4 

* 

1 

24 

3.57 

85.71 

5 

*** 

3 

27 

10.71 

96.43 

7 

* 

1 

28 

3.57 

100.00 

+ - + - + - + - +- 


4  8  12  16  20 

FREQUENCY 


Phrases 

1 

2 

3 

4 

5 

6 
7 


Use  your  common  sense 

Allot  a  specific  amount  of  time  to  each  task 
Tolerate  Uncertainty 

Handle  ambiguity  by  developing  opposing  points  of  view 
Expect  that  a  task  will  be  transformed  while  it  is 
in  progress 

Transform  the  task  to  create  uncertainty 
(Two  or  more  phrases  identified  as  "Most") 
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TzUsle  8  Continued 


LEVEL»HiIV 


FREQUENCY  OF  SET  F 


CUM 

CUM 

1 

FREQ 

FREQ 

PERCENT 

PERCENT 

1 

j  ***************** 

17 

17 

70.83 

70.83 

2 

* 

1 

18 

4.17 

75.00 

3 

** 

2 

20 

8.33 

83.33 

4 

* 

1 

21 

4.17 

87.50 

5 

* 

1 

22 

4.17 

91.67 

7 

** 

2 

24 

8.33 

100.00 

•+ + + - +- 

4  8  12  16 

FREQUENCY 


LEVEL=V 


FREQUENCY  OF  SET  F 


1 

3 

5 

6 
7 


CUM  CUM 

FREQ  FREQ  PERCENT  PERCENT 


I 

********** 

j  ************************* 
********** 

j  ********** 

- + - + - + - + - + 

1  2  3  4  5 

FREQUENCY 


2 

2 

15.38 

15.38 

2 

4 

15.38 

30.77 

5 

9 

38.46 

69.23 

2 

11 

15.38 

84.62 

2 

13 

15.38 

100.00 

Table  9 


■♦Most"  Responses  to  Set  G  of  CPA  PHRASES  by  Subjects  Classified 
at  Four  Different  Current  Capacity  Levels  (ill.  LqIV.  Hiiv.  and 

Y1 


LEVEL=III 


FREQUENCY  OF  SET  G 


CUM 

CUM 

1 

FREQ 

FREQ 

PERCENT 

PERCENT 

1 

j  *it***it*** 

3 

3 

10.00 

10.00 

2 

********* 

3 

6 

10.00 

20.00 

3 

********* 

3 

9 

10.00 

30.00 

4 

*************************** 

9 

18 

30.00 

60.00 

5 

j  *************** 

5 

23 

16.67 

76.67 

6 

j  ********************* 

7 

30 

23.33 

100.00 

■+ - + - + - + - + - + - + - + - 4 


123456789 

FREQUENCY 

LEVEL=LoIV 

FREQUENCY  OF  SET  G 


1 

FREQ 

CUM 

FREQ 

PERCENT 

CUM 

PERCENT 

1 

j  ** 

1 

1 

3.57 

3.57 

3 

**** 

2 

3 

7.14 

10.71 

4 

'if’kitititifiticitifitltititieieiBitititit'itieitltltiUt’klt  15 

18 

53.57 

64.29 

5 

** 

1 

19 

3.57 

67.86 

6 

j  ************** 

7 

26 

25.00 

92.86 

7 

**** 

2 

28 

7.14 

100.00 

4- - 4- - 4- - 4- - 4- - 4- - 4— 

2  4  6  8  10  12  14 

FREQUENCY 


Phrases  - 

1  -  Stop  if  there  is  a  problem 

2  -  See  gaps  in  knowledge  as  interruptions  to  work 

3  -  See  gaps  in  knowledge  as  missing  links  in  a  chain 

4  -  See  gaps  in  knowledge  as  missing  pieces  of  a  jigsaw 

5  -  See  gaps  as  the  most  interesting  part 

6  -  See  gaps  as  pauses  in  the  process 

7  -  (Two  or  more  phrases  identified  as  "Most") 
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Table  9  Continued 


LEVEL*HiIV 


FREQUENCY  OF  SET  G 


CUM 

CTJM 

1 

FREQ 

FREQ 

PERCENT 

PERCENT 

1 

■k* 

1 

1 

4.17 

4.17 

2 

** 

1 

2 

4.17 

8.33 

4 

****************** 

9 

11 

37.50 

45.83 

5 

J  ************** 

7 

18 

29.17 

75.00 

6 

******** 

4 

22 

16.67 

91.67 

7 

j  **** 

2 

24 

8.33 

100.00 

■+ + + + + 

1  3  5  7  9 

FREQUENCY 


LEVE1>V 


FREQUENCY 

OF 

SET  G 

CUM 

CUM 

FREQ 

1 

FREQ 

PERCENT 

PERCENT 

3 

1** 

1 

1 

1 

7.69 

7.69 

4 

** 

1 

2 

7.69 

15.38 

5 

****************** 

9 

11 

69.23 

84.62 

7 

j  **** 

1  3  5  7  9 

FREQUENCY 

2 

13 

15.38 

100.00 
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Table  10 


"Most"  Responses  to  Set  H  of  CPA  PHRASES  bv  Subjects  Classified 
at  Four  Different  Current  Capacity  Levels  fill.  LoIV.  HilV.  and 

Y1 


LEVEL*III 


FREQUENCY  OF  SET  H 


1 

FREQ 

CUM 

FREQ 

PERCENT 

CUM 

PERCENT 

1 

j  ** 

1 

1 

3.23 

3.23 

2 

***************** 

•!t  ********  13 

14 

41.94 

45.16 

3 

************** 

7 

21 

22.58 

67.74 

4 

j  ********** 

5 

26 

16.13 

83.87 

5 

j  ********** 

5 

31 

16.13 

100.00 

- + - + - + - + - + - +- 

2  4  6  8  10  12 

FREQUENCY 

LEVEL=LoIV 


FREQUENCY  OF  SET  H 


1 

FREQ 

CUM 

FREQ 

PERCENT 

CUM 

PERCENT 

2 

**************** 

8 

8 

28.57 

28.57 

3 

j  ************** 

7 

15 

25.00 

53.57 

4 

******************** 

10 

25 

35.71 

89.29 

5 

**** 

2 

27 

7.14 

96.43 

7 

** 

1 

28 

3.57 

100.00 

+ - + - + - + - + - + 


2  4  6  8  10 

FREQUENCY 


Phrases  - 

1  -  Expect  to  be  told  what  to  do 

2  -  Work  out  the  answer  from  previous  experience 

3  -  Expect  that  a  solution  will  emerge 

4  -  Resolve  tasks  by  choosing  between  alternatives 

5  -  Seek  original  solutions 

6  -  See  the  solution  as  the  beginning  of  a  new  problem 

7  -  (Two  or  more  phrases  identified  as  "Most") 
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Table  10  Continued 


LEVEL=HiIV 


FREQUENCY  OF  SET  H 


CUM 

CUM 

1 

FREQ 

FREQ 

PERCENT 

PERCENT 

2 

j  ************************* 

5 

5 

20.83 

20.83 

3 

****************************** 

6 

11 

25.00 

45.83 

4 

j  ************************* 

5 

16 

20.83 

66.67 

5 

I  ******************** 

4 

20 

16.67 

83.33 

6 

***** 

1 

21 

4.17 

87.50 

7 

*************** 

3 

24 

12.50 

100.00 

,+ - + - + - + - + - + 

1  2  3  4  5  6 

FREQUENCY 


LEVEL=V 


FREQUENCY 

OF  SET  H 

CUM 

CUM 

1 

FREQ 

FREQ 

PERCENT 

PERCENT 

2 

j  ********** 

1 

1 

8.33 

8.33 

3 

******************** 

2 

3 

16.67 

25.00 

4 

j  ******************** 

2 

5 

16.67 

41.67 

5 

j  ifkisisigititititltititititititifkitit'kifkifftfiitititlt  3 

8 

25.00 

66.67 

6 

******************** 

2 

10 

16.67 

83.33 

7 

******************** 

2 

12 

16.67 

100.00 

.+ - + - + 

12  3 

FREQUENCY 
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Table  11 


"Most"  Responses  to  Set  I  of  CPA  PHRASES  bv  Subjects  Classified 
at  Four  Different  Current  Capacity  Levels  fill.  LoIV.  HilV.  and 

VI 


LEVEL=III 


FREQUENCY  OF  SET  I 


1 

FREQ 

CUM 

FREQ 

PERCENT 

CUM 

PERCENT 

1 

j  ***************** 

17 

17 

54.84 

54.84 

2 

***** 

5 

22 

16.13 

70.97 

3 

***** 

5 

27 

16.13 

87.10 

4 

* 

1 

28 

3.23 

90.32 

6 

** 

2 

30 

6.45 

96.77 

7 

* 

1 

31 

3.23 

100.00 

+ - + - + - + - + 


4  8  12  16 

FREQUENCY 

LEVEL=LoIV 


FREQUENCY  OF  SET  I 


FREQ 

1 

CUM 

FREQ 

PERCENT 

CUM 

PERCENT 

1 

j  **************************** 

7 

7 

25.00 

25.00 

2 

j  ******************** 

5 

12 

17.86 

42.86 

3 

******** 

2 

14 

7.14 

50.00 

4 

**************************** 

7 

21 

25.00 

75.00 

5 

j  ******** 

2 

23 

7.14 

82.14 

6 

j  **************** 

4 

27 

14.29 

96.43 

7 

**** 

1 

28 

3.57 

100.00 

.+ - + - + - + - + - + - + 

1  2  3  4  5  6  7 

FREQUENCY 


Phrases  - 

1  -  Answers  should  be  straightforward 

2  -  Options  should  not  be  discarded 

3  -  Go  back  to  the  beginning  if  the  thread  is  lost 

4  -  Hold  a  solution  while  developing  an  alternative  approach 

5  -  Discard  solutions  when  you  deem  it  necessary 

6  -  There  are  no  permanent  solutions 

7  -  (Two  or  more  phrases  identified  as  "Most") 
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^  to  H 


Table  11  Continued 


LEVEL*HiIV 


FREQUENCY  OF  SET  I 


FREQ 

CUM 

FREQ 

PERCENT 

CUM 

PERCENT 

9 

9 

37.50 

37.50 

3 

12 

12.50 

50.00 

4 

16 

16.67 

66.67 

6 

22 

25.00 

91.67 

■klfkltlfk 

2 

24 

8.33 

100.00 

.+ — + — + — + — + — + — + — + — + 


123456789 

FREQUENCY 


LEVED=V 


FREQUENCY  OF  SET  I 


1 

FREQ 

CUM 

FREQ 

PERCENT 

CUM 

PERCENT 

1 

**** 

2 

2 

15.38 

15.38 

6 

******************** 

10 

12 

76.92 

92.31 

7 

** 

1 

13 

7.69 

100.00 

■+ - + - + - + - + 


2  4  6  8  10 

FREQUENCY 
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APPENDIX  B 


Revised  PHRASES  Item  Set  and 
Scoring  Key 


B-1 


WORK  STYLE  IMVEliTORY 


On  each  of  the  following  pages  you  will  find  a  list  of  six 
phrases.  Each  phrase  describes  one  way  of  approaching  work. 

Your  first  task  is  to  read  each  of  the  six  phrases  and  then 
choose  the  one  which  is  MOST  LIKE  THE  WAY  YOU  PREFER  TO  APPROACH 
A  PIECE  OF  WORK.  When  you  have  decided,  place  an  "N”  to  the  left 
of  that  phrase. 

There  are  no  right  or  wrong  answers.  We  are  only  interested  in 
how  you  prefer  to  operate  when  working. 

After  you  have  indicated  which  phrase  is  most  like  you  prefer  to 
approach  a  piece  of  work,  next  decide  which  of  the  remaining 
phrases  is  LEAST  LIKE  THE  WAY  YOU  PREFER  TO  APPROACH  YOUR  WORK. 
When  you  have  decided,  place  a  "L"  to  the  left  of  that  phrase. 

Finally,  at  the  bottom  of  each  page  write  two  or  three  sentences 
indicating  whv  you  picked  the  phrase  you  did  as  your  most 
preferred  phrase. 


IF  YOU  ARE  AT  ALL  UNCLEAR  ABOUT  HOW  TO  PROCEED,  PLEASE  ASK  US  TO 
CLARIFY  THESE  INSTRUCTIONS 


B-2 


Place  an  "M**  to  the  left  of  the  phrase  which  is  most  like  the  way 
you  prefer  to  approach  a  piece  of  work 

Next  place  an  "L"  to  the  left  of  the  phrase  which  is  least  like 
the  way  you  prefer  to  approach  your  work 

Set  A: 


Work  within  a  given  framework 

Work  with  a  minimum  of  preconceptions 

Define  the  horizons  of  the  work 

Work  to  a  complete  set  of  instructions 

Work  with  connections  even  if  particular  links  are 
unclear 

_  Work  in  abstracts  and  concepts 


Below  write  a  few  sentences  which  explain  whv  you  picked  the 
particular  phrase  you  did  as  your  "MOST”  choice. 


B-3 


Place  an  "M"  to  the  left  of  the  phrase  which  is  most  like  the  way 
you  prefer  to  approach  a  piece  of  work 

Next  place  an  ”L"  to  the  left  of  the  phrase  which  is  least  like 
the  way  you  prefer  to  approach  your  work 

Set  B: 


Use  words,  ideas,  and  theories  as  tools 

Establish  new  relationships  between  previously 
unrelated  materials 

Do  one  thing  at  a  time 

Co-ordinate  by  drawing  together  a  number  of  separate 
strands 

Focus  on  one  part  of  the  task  at  a  time 
Compare  the  merits  of  alternative  options 


Below  write  a  few  sentences  which  explain  why  you  picked  the 
particular  phrase  you  did  as  your  "MOST”  choice. 


B-4 


Place  an  "ll'*  to  the  left  of  the  phrase  which  is  most  like  the  way 
you  prefer  to  approach  a  piece  of  work 

Next  place  an  "L”  to  the  left  of  the  phrase  which  is  least  like 
the  way  you  prefer  to  approach  your  work 

SET  C: 

_  Follow  the  rules 

_  Make  sure  the  rules  fit 

_  Redefine  the  rules 

_  Look  for  the  intent  of  the  rules 

_  Work  within  the  rules 

_  Use  the  rules  as  guides  to  action 


Below  write  a  few  sentences  which  explain  whv  you  picked  the 
particular  phrase  you  did  as  your  “MOST"  choice. 


B-5 


Place  an  "N**  to  the  left  of  the  phrase  which  is  most  like  the  way 
you  prefer  to  approach  a  piece  of  work 

Next  place  an  "L"  to  the  left  of  the  phrase  which  is  least  like 
the  way  you  prefer  to  approach  your  work 

SET  D: 

_  Take  a  systematic  approach 

_  Look  for  relationships  between  the  current  task  and 

other  tasks 

_  Approach  each  task  in  its  own  right 

_  Transcend  the  task 

_  Follow  instructions  carefully 

_  Span  a  broad  spectrum  and  also  focus  in  detail  on 

certain  aspects 

Below  write  a  few  sentences  which  explain  why  you  picked  the 
particular  phrase  you  did  as  your  "MOST"  choice. 


B-6 


Place  an  "M"  to  the  left  of  the  phrase  which  is  most  like  the  way 
you  prefer  to  approach  a  piece  of  work 

Next  place  an  "L"  to  the  left  of  the  phrase  which  is  least  like 
the  way  you  prefer  to  approach  your  work 

SET  E: 

_  Look  for  underlying  issues 

_  Break  up  the  problem  into  separate  parts 

_  Consider  the  context  of  the  problem 

_  Do  first  things  first 

_  Look  for  underlying  issues 

_  Figure  out  the  right  sequence  of  tasks 

Below  write  a  few  sentences  which  explain  whv  you  picked  the 
particular  phrase  you  did  as  your  "MOST”  choice. 


B-7 


Place  an  "M"  to  the  left  of  the  phrase  which  is  most  like  the  way 
you  prefer  to  approach  a  piece  of  work 

Next  place  an  "L"  to  the  left  of  the  phrase  which  is  least  like 
the  way  you  prefer  to  approach  your  work 

SET  F: 


Handle  ambiguity  by  developing  opposing  points  of 
view 

Follow  a  set  procedure 

Allot  a  specific  amount  of  time  to  each  task 
Transform  the  task 

Expect  that  a  task  will  be  transformed  while  it  is  in 
progress 

Impose  a  procedure  to  reduce  uncertainty 


Below  write  a  few  sentences  which  explain  why  you  picked  the 
particular  phrase  you  did  as  your  "MOST”  choice. 


Place  an  "M”  to  the  left  of  the  phrase  which  is  most  like  the  way 
you  prefer  to  approach  a  piece  of  work 

Next  place  an  "L"  to  the  left  of  the  phrase  which  is  least  like 
the  way  you  prefer  to  approach  your  work 

SET  G: 

_  See  gaps  in  knowledge  as  missing  pieces  of  a  jigsaw 

_  See  gaps  as  pauses  in  the  process 

_  See  gaps  in  knowledge  as  interruptions  to  work 

_  Know  that  new  information  creates  new  gaps 

_  Stop  if  there  is  a  problem 

_  See  gaps  as  the  most  interesting  part 


Below  write  a  few  sentences  which  explain  why  you  picked  the 
particular  phrase  you  did  as  your  "MOST”  choice. 


t 


Place  an  ”M"  to  the  left  of  the  phrase  which  is  most  like  the  way 
you  prefer  to  approach  a  piece  of  work 

Next  place  an  ”L”  to  the  left  of  the  phrase  which  is  least  like 
the  way  you  prefer  to  approach  your  work 

SET  H; 


Expect  to  be  told  what  to  do 

Resolve  tasks  by  formulating  alternatives 

See  the  solution  as  the  beginning  of  a  new  problem 

Rely  mainly  on  previous  experience 

Expect  that  the  situation  will  resolve  itself  in  time 
Seek  to  develop  an  original  solution 


Below  write  a  few  sentences  which  explain  whv  you  picked  the 
particular  phrase  you  did  as  your  ’’MOST”  choice . 
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Place  an  "N"  to  the  left  of  the  phrase  which  is  most  like  the  way 
you  prefer  to  approach  a  piece  of  work 

Next  place  an  "L"  to  the  left  of  the  phrase  which  is  least  like 
the  way  you  prefer  to  approach  your  work 

SET  I: 

_  There  are  no  permanent  solutions 

_  Most  problem  solutions  are  straightforward 

_  Options  should  not  be  discarded 

_  Go  back  to  the  beginning  if  the  thread  is  lost 

_  Conflicting  solutions  must  sometimes  be  accepted 

_  Hold  a  solution  while  developing  an  alternative 

approach 


Below  write  a  few  sentences  which  explain  why  you  picked  the 
particular  phrase  you  did  as  your  "MOST”  choice. 
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REVISED  PHRASES  (PHRASES  II) 
SCORING  KEY  (2/93) 


SET  A: 

II  Work  within  a  given  framework 
V  Work  with  a  minimum  of  preconceptions 
VI  Define  the  horizons  of  the  work 
I  Work  to  a  complete  set  of  instructions 
III  Work  with  connections  even  if  particular  links  are 
unclear 

_ IV  Work  in  abstracts  and  concepts 


SET  B: 

VI 

_ V 


_ I 

III 

II 

_IV 


Use  words,  ideas,  and  theories  as  tools 
Establish  new  relationships  between  previously 
unrelated  materials 
Do  one  thing  at  a  time 

Co-ordinate  by  drawing  together  a  number  of  separate 
strands 

Focus  on  one  part  of  the  task  at  a  time 
Compare  the  merits  of  alternative  options 


SET  C; 

_ I 

III 
VI 
IV 
1.1 
_ V 


Follow  the  rules 

Make  sure  the  rules  fit 

Redefine  the  rules 

Look  for  the  intent  of  the  rules 

Work  within  the  rules 

Use  the  rules  as  guides  to  action 


SET  D: 

III 

_ V 


II. 

VI 

I 

IV 


Take  a  systematic  approach 

Look  for  relationships  between  the  current  task  and 
other  tasks 

Approach  each  task  in  its  own  right 

Transcend  the  task 

Follow  instructions  carefully 

Span  a  broad  spectrum  and  also  focus  in  detail  on 
certain  aspects 


SET  E: 

_ V 

_ II 

VI 

_ I 

IV 

III 


Look  for  underlying  issues 
Break  up  the  problem  into  separate  parts 
Consider  the  context  of  the  problem 
Do  first  things  first 

Create  an  overall  picture  of  the  problem 
Figure  out  the  right  sequence  of  tasks 
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SET  F; 


V 


IV  Handle  ambiguity  by  developing  opposing  points  of 
view 

I  Follow  a  set  procedure 
II  Allot  a  specific  amount  of  time  to  each  task 
VI  Transform  the  task 

V  Expect  that  a  task  will  be  transformed  while  it  is  in 

progress 

III  Impose  a  procedure  to  reduce  uncertainty 

SET  G: 

IV  See  gaps  in  knowledge  as  missing  pieces  of  a  jigsaw 

III  See  gaps  as  pauses  in  the  process 

II  See  gaps  in  knowledge  as  interruptions  to  work 

VI  Know  that  new  information  creates  new  gaps 
I  Stop  if  there  is  a  problem 

V  See  gaps  as  the  most  interesting  part 

SET  H: 

I  Expect  to  be  told  what  to  do 
IV  Resolve  tasks  by  fontulating  alternatives 
VI  See  the  solution  as  the  beginning  of  a  new  problem 
II  Rely  mainly  on  previous  experience 
III  Expect  that  the  situation  will  resolve  itself  in  time 

V  Seek  to  develop  an  original  solution 


SET  I: 

V  There  are  no  permanent  solutions 
I  Most  problem  solutions  are  straightforward 
II  Options  should  not  be  discarded 
III  Go  back  to  the  beginning  if  the  thread  is  lost 
VI  Conflicting  solutions  must  sometimes  be  accepted 
IV  Hold  a  solution  while  developing  an  alternative 
approach 
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